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Abstract — Despite  “airborne  network  (AN)  topology  de¬ 
sign”  and  careful  planning  of  AN  trajectories,  unexpected 
disruptions  (from  hardware  failures  to  changes  in  mission 
requirements  and  hostile  attacks)  may  cause  nodes  not  to 
connect  to  one  another  directly  or  indirectly  either  because 
they  are  out  of  one  another’s  range  or  because  nodes  do  not 
meet  one  another  according  to  their  preplanned  trajecto¬ 
ries.  Since  an  end-to-end  path  within  the  AN  is  not  always 
guaranteed,  packets  have  to  be  delivered  in  a  delay-tolerant 
fashion,  namely,  some  intermediate  nodes  will  need  to 
buffer  packets  during  times  of  disconnectivity.  In  our  ear¬ 
lier  work  we  developed  Mobility  Aware  Routing  Protocol 
and  Mobility  Dissemination  Protocol  (MARP/MDP)  that 
used  preplanned  trajectories  of  airborne  nodes  to  make 
intelligent  routing  decisions  preemptively.  In  this  paper  we 
present  a  delay-tolerant  strategy  (MARP/MDP+DTN)  to 
predict  the  minimum  end-to-end  delay  and  obtain  the  cor¬ 
responding  path.  In  addition,  MARP/MDP+DTN  accounts 
for  local  queueing  (MARP/MDP+DTN+QC)  to  minimize 
congestion  and  further  improves  end-to-end  delay  with  the 
positive  side  effect  of  load-balancing.  Simulation  results 
have  shown  an  improvement  of  52%  in  packet  delivery 
ratio  in  MARP/MDP+DTN.  MARP/MDP+DTN+QC  also 
exhibits  extremely  short  latency,  about  90%  reduction 
from  MARP/MDP+DTN  in  highly  congested  network. 
Moreover,  MARP+DTN+QC  balances  local  traffic  67% 
better  than  MARP+DTN  in  high  traffic  load  scenarios. 

I.  Introduction 

The  Airborne  Network  (AN)  will  form  an  essential 
part  of  the  Global  Information  Grid  in  the  future,  thus 
providing  information  and  decision  superiority  to  US 
armed  forces.  AN,  an  enabling  technology  for  Network 
Centric  Warfare,  differs  from  the  terrestrial  mobile  ad- 
hoc  networks  (MANETs)  and  the  wire-line  Internet,  both 
in  terms  of  network  capability  and  underlying  assump¬ 
tions.  For  one,  the  AN  backbone  nodes  which  provide 
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almost-permanent  connectivity  are  envisioned  to  fly  in 
pre-planned  orbits.  Moreover,  the  transmitter  power  of 
these  nodes  are  significantly  higher  than  battery  operated 
MANET  nodes. 

Mobility  Aware  Routing  Protocol/Mobility  Dissem¬ 
ination  Protocol  (MARP/MDP)  protocol  suite  [4] 
is  designed  specifically  for  the  airborne  networks. 
MARP/MDP  routes  traffic  based  on  the  knowledge  of 
planned  nodes’  trajectories  with  respect  to  time  and 
makes  preemptive  decisions  to  minimize  packet  losses 
due  to  link  failure  and  discover  better  routes.  For  every 
topology  entry,  MARP  computes  Dijktras  shortest  paths 
between  itself  and  every  receiver  and  stores  this  infor¬ 
mation  in  a  T-time  routing  table,  whose  routing  entries 
change  as  a  function  of  time.  In  MARP,  therefore,  a 
successful  route  lookup  is  the  output  of  the  current  time 
instant  and  the  destination’s  address.  Since  it  is  possible 
that  some  airborne  platforms  may  not  adhere  to  the 
pre-decided  flight  plans  or  links  go  down  unexpectedly 
because  of  weather  conditions  and  link  quality,  MDP  is 
used  to  inform  all  network  nodes  of  any  deviation  from 
the  preplanned  behavior  so  that  their  T-time  routing  table 
can  be  updated  accordingly. 

In  spite  of  “airborne  network  (AN)  topology  de¬ 
sign”  and  careful  planning  of  AN  trajectories  by 
MARP/MDP,  unexpected  disruptions  (from  hardware 
failures  to  changes  in  mission  requirements  and  hostile 
attacks)  may  cause  nodes  not  to  connect  to  one  another 
at  all  directly  or  indirectly  either  because  they  are  out 
of  one  another’s  range  or  because  nodes  do  not  meet 
one  another  according  to  their  preplanned  trajectories. 
Since  an  end-to-end  path  within  the  AN  is  not  always 
guaranteed,  packets  have  to  be  delivered  in  a  delay- 
tolerant  fashion,  namely,  some  intermediate  nodes  will 
need  to  buffer  packets  during  times  of  disconnectivity. 
This  delay  tolerant  operation  assumes  that  the  appli¬ 
cations  routed  on  these  paths  tolerate  the  associated 
delay.  With  the  preplanned  trajectories  of  nodes  in  an 


Report  Documentation  Page 


Form  Approved 
0MB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 


1.  REPORT  DATE 

OCT  2009 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2009  to  00-00-2009 


4.  TITLE  AND  SUBTITLE 

Delay  Tolerant  Mobility  Aware  Routing/Mobility  Dissemination  Protocol 
for  the  Airborne  Network 

6.  AUTHOR(S) 


7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

University  of  California,  Los  Angeles, Department  of  Computer 
Science, Los  Angeles, CA, 90095 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

MILCOM  2009,  Boston,  MA,  October.  2009 

14.  ABSTRACT 

see  report 

15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OE  PAGES 

19a.  NAME  OE 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

7 

Standard  Form  298  (Rev.  8-98} 

Prescribed  by  ANSI  Std  Z39-18 


airborne  network  (AN),  a  delay-tolerant  strategy  can  be 
obtained  to  predict  the  minimum  delay  and  to  quantify 
resource  (buffer)  usage  for  any  set  of  “failures”  or  of 
“known  departures”  from  the  initial  schedules.  In  the 
face  of  an  Unattended  Airborne  Vehicle  (UAV)  failure, 
this  prediction  is  used  to  determine  whether  the  AN 
satisfies  the  application  requirement.  Efficient  resource 
usage  is  also  important  to  ensure  that  there  is  enough 
buffer  space  for  data  communication  on  ever-improving 
link  technology.  By  integrating  knowledge  of  trajectories 
of  the  AN  backbone,  we  can  quantify  buffer  space,  route 
more  intelligently,  and  thereby  reduce  packet  failure  rate 
and  efficiently  manage  network  resources. 

In  this  paper,  we  propose  MARP/MDP-i-DTN  that  en¬ 
ables  AN  routing  in  the  presence  of  intermittent  connec¬ 
tivity.  The  contributions  of  the  paper  are  two-fold.  First, 
MARP/MDP-i-DTN  is  proposed  to  provide  the  best  next 
forwarding  hop  regardless  of  current  network  condition. 
Unlike  MARP/MDP,  MARP/MDP-i-DTN  tolerates  delay 
and  therefore  provide  better  end-to-end  packet  delivery. 
Second,  MARP/MDP-i-DTN  is  traffic-aware.  DTN  fea¬ 
ture  comes  at  a  cost  of  queueing.  To  minimize  delay 
that  packets  experience  in  queuing,  MARP/MDP-i-DTN 
is  made  sensitive  to  queuing.  It  routes  traffic  around 
congested  network  region. 

The  remaining  of  the  paper  is  organized  as  follows: 
Section  II  provides  our  delay  tolerant  strategy  which 
exploits  the  AN  node  preplanned  trajectories.  Section  III 
considers  queueing  condition  at  each  node  to  further 
optimize  MARP/MDP-i-DTN  by  reducing  end-to-end  de¬ 
lay.  Section  IV  evaluates  MARP/MDP-i-DTN  with  pure 
MARP/MDP  in  different  connectivity  scenarios  and  dif¬ 
ferent  queueing  loads.  Section  V  concludes  the  paper 
and  presents  future  plan  for  MARP/MDP-i-DTN. 

II.  DTN  WITH  Modified  Dijkstra  Algorithm 

The  AN  belongs  to  the  forwarding-based  DTN  ap¬ 
proach  since  the  preplanned  trajectories  are  known.  Fur¬ 
thermore,  we  can  think  of  the  topology  as  a  time- varying 
graph  [2].  As  time  progresses,  links  between  nodes  either 
go  up  or  down,  forming  a  particular  graph  in  that  time 
duration.  According  to  [3],  a  modified  Dijkstra’s  algo¬ 
rithm  with  time-varying  edge  costs  can  be  constructed  to 
predict  the  minimum  delay  between  nodes  and  quantify 
the  buffer  space  at  each  node.  MARP/MDP-i-DTN  uses 
the  modified  Dijkstra’s  algorithm  that  accounts  for  link 
duration  to  compute  the  next  forwarding  hop  which 
yields  minimum  end-to-end  delay.  When  there  is  no  end- 
to-end  connectivity,  MARP/MDP-i-DTN  would  compute 
the  best  next  forwarding  hop  to  minimize  end-to-end 
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Fig.  I.  4-node  time- varying  topologies,  single-encountering. 


delay.  Each  node  is  equipped  with  a  queue  to  store  the 
packets  when  appropriate. 


Algorithm  1  Modified  Dijkstra 

Require:  G  =  {V,  E),  s,  T,  w{e,  t) 
l  :  Q  V 

2:  L[s]  4=  0,  L[v]  <1=  oo  V  V  G  U  s.t.  v  ^  s 
3:  while  <5  /  0  do 

4:  u  X  G  Q  S.t.  L[x]  =  miny^QL[y] 

5:  Q  <=  Q\{u} 

6:  for  all  e  G  £■  s.t.  e  =  (tt,  v)  do 

7:  if  L[v]  >  {L[u]  -f  w{e,  L[u]  +  T))  then 

8:  L[v]  4=  L[u\  +  vj{e,  L[u\  +  T) 

9:  end  if 

10:  end  for 

II:  end  while 


Initially,  a  time-varying  graph  G  =  {V,  E),  a  source 
node  s,  the  start  time  T,  and  the  cost  of  an  edge  at  time 
t,  w{e,  t)  are  given.  The  modified  Dijkstra’s  algorithm 
outputs  the  path  with  the  minimum  delay  in  an  array  L. 
L[x]  indicates  the  cost  of  going  to  x  from  s.  In  Fine  1, 
a  set  Q  that  contains  all  the  vertices  in  V  is  created. 
The  cost  of  source  node  L[s],  that  is  the  cost  of  going 
to  itself,  is  assigned  to  0;  the  cost  of  the  rest  of  nodes 
is  assigned  to  infinity  in  Fine  2.  From  Fine  3  to  11, 
there  is  a  while  loop  that  terminates  when  Q  becomes 
the  empty  set.  While  Q  is  not  empty,  a  vertex  u  that  has 
the  minimum  cost  is  picked  from  Q  in  Fine  4.  u  is  then 
removed  from  the  set  Q  in  Fine  5.  For  a  vertex  v  that 
forms  an  edge  with  u  in  Fine  6,  the  algorithm  checks 
if  the  current  cost  of  v,  L[v],  is  greater  than  the  sum  of 
the  cost  of  going  to  u  and  the  cost  of  going  from  u  to 
V,  given  that  the  cost  of  u,  L[u],  and  the  start  time  T  in 
Fine  7.  If  it  is,  L[v]  is  assigned  to  the  sum  of  the  cost 
of  going  to  u  and  the  cost  of  going  from  u  to  u  at  time 
{L[u]  +  T)  in  Fine  8.  When  the  algorithm  finishes,  the 
array  L  will  have  the  minimum  delay  from  s  to  each 
node  in  the  network. 

Figure  1  shows  an  example  of  4-node  time-varying 
topologies  where  the  nodes  in  the  time-varying  graphs 
are  UAVs  flying  in  their  orbit.  Finks  between  UAVs 
change  when  they  fly  away  from  each  other  in  their 
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orbits.  In  this  particular  example,  Node  1  is  connected 
to  Node  2  and  4  for  3  seconds.  Node  2  and  Node  4  are 
connected  for  the  next  second.  Node  1  and  Node  3  are 
connected  the  next  second.  Finally,  Node  1  is  connected 
to  Node  4  and  Node  2  to  Node  3  for  one  second. 

Table  I  shows  modified  Dijkstra’s  computation  of 
minimum  delay  (L)  from  Node  1  to  the  other  three 
nodes  and  their  predecessor.  Each  row  represents  one 
while  loop  run  in  Line  3  of  the  modified  Djikstra’s 
algorithm.  The  last  row  is  node’s  minimum  delay  and  its 
predecessor.  For  example,  it  takes  Node  1  a  minimum  of 
5  seconds  to  reach  Node  3,  whose  previous  hop  is  Node 
1. 

Note  that  when  Q  contains  {3,4},  L[3]  <  1  + 
te(e(2,3),l)  =  1  +  (4  +  1).  m(e(2,3),l)  indicates 
the  cost  to  reach  Node  3  from  Node  2  at  time  1. 
m(e(2,  3),  1)  =4  +  1  because  it  takes  4  seconds  for  the 
edge  between  Node  2  and  Node  3  to  become  available 
since  time  1  and  it  takes  1  second  for  the  packet  to  be 
sent  from  Node  2  to  Node  3.  This  example,  based  on 
nodes’  predecessor,  shows  that  Node  1  can  reach  Node 
2  starting  at  0  second.  Node  1  can  reach  Node  3  starting 
at  4th  second.  Node  1  can  reach  Node  4  starting  at  0 
second. 

When  airborne  nodes  deviate  from  their  planned 
trajectory  in  the  event  of  adverse  weather  conditions 
or  emergency.  Mobility  Dissemination  Protocol  (MDP) 
Discrepancy  packets  are  flooded  in  the  network  to  update 
each  node’s  routing  table  accordingly  (See  [4]  for  detail). 
As  we  have  seen  before,  MARP  predicted  topology 
is  a  piecewise  constant  function  of  time.  MDP  link 
state  updates  do  not  contain  time  to  live,  therefore 
MARP/MDP+DTN  assumes  that  a  Discrepancy  indi¬ 
cated  by  MDP  is  only  valid  till  the  future  time  instant 
when  the  predicted  topology  changes.  Then  the  modified 
Dijkstra’s  algorithm  is  called  on  the  updated  time  depen¬ 
dent  topology  to  yield  the  best  route  to  the  destination. 

III.  Local  Queuing  Aware  Scheduling 

When  disconnectivity  increases,  there  will  be  large 
queuing  delay.  To  minimize  it,  we  account  for  queuing  at 
all  edges  outgoing  from  the  current  node  to  route  around 
congestion  at  the  first  hop.  The  path  traversed  thus  is 
sensitive  to  the  queuing  present  at  all  edges  in  the  path. 
Because  queuing  information  is  local,  we  recompute  the 
route  at  every  hop  (per-hop  routing). 

During  each  periodic  beaconing,  each  MARP/MDP 
node  will  forward  its  queue  size  to  its  neighbors.  Note 
that  receiving  the  queue  size  of  its  neighbors  is  just  as 
reliable  as  receiving  hello  messages.  Unless  the  neighbor 


is  out  of  the  node’s  radio  range,  a  hello  message  along 
with  the  queue  size  will  be  received  the  next  beacon 
period.  The  modified  Dijkstra  will  select  the  next  hop 
to  each  destination  with  the  minimum  delay  based  on 
the  next  hop’s  queue  size.  “L[u]  +  w{e,L[u]  +  T)”  in 
Lines  7  and  8  of  the  modified  Dijkstra  will  reflect  the 
current  queue  size  of  w,  specifically,  w{e,L[u]  +T)  will 
incorporate  the  cost  of  sending  packets  already  in  the 
queue  at  u,  in  addition  to  the  one  packet  that  is  scheduled 
to  be  sent  from  u.  Packets  are  dequeued  on  the  first-in 
first-out  basis.  Although  each  packet  in  the  queue  may 
be  going  through  a  different  neighbor  towards  a  different 
destination,  we  assume  that  they  are  going  to  the  same 
destination  v  as  the  last  packet.  The  calculation  gives  a 
conservative  and  upperbound  cost  to  v  through  u.  The 
cost  estimation  based  on  the  queue  will  avoid  nodes  with 
large  queue  in  general  as  each  packet  is  subject  to  the 
same  delay  (i.e.,  the  period  when  (u,  v)  is  up  is  the 
same).  The  slight  difference  in  queue  size  between  two 
nodes  may  favor  the  one  smaller  even  though  packets 
in  the  larger  queue  are  dequeued  faster  because  many 
of  them  are  going  through  links  that  come  up  earlier. 
We  show  the  computation  of  the  link  cost  in  single  and 
multiple  encountering  in  the  following  examples. 

A.  Single  Encountering 

The  example  follows  the  same  time-varying  graph  as 
Figure  1.  However,  assume  that  there  are  two  messages 
in  Node  2’s  queue  for  the  sake  of  illustration  of  how 
MARP/MDP-i-DTN  with  queueing  awareness  enabled 
would  react  differently  to  traffic  congestion.  Further¬ 
more,  assume  that  the  transmission  rate  is  1  msg/sec' 
and  propagation  delay  is  negligible.  Table  II  shows  the 
table  of  modified  Dijkstra’s  computation  from  Node  1 
to  the  other  three  nodes.  At  the  last  row,  the  minimum 
delay  along  with  the  node’s  predecessor  is  computed. 

Note  that  when  Q  contains  {3,4},  L[3]  is  compared 
with  1  +  w{e{2,  3),  1)  =  1  +  5  +  6  +  6.  w{e{2, 3),  1)  = 
5  +  6  +  6  because  it  takes  5  more  seconds  to  dequeue 
the  first  packet,  6  seconds  to  dequeue  the  second  packet, 
and  another  6  seconds  to  dequeue  the  last  packet.  L[4] 
is  compared  with  1  +  rt;(e(2,4),  1)  =  1  +  3  +  6  + 6. 
m(e(2, 4),  1)  =  3  +  6  +  6  because  it  takes  3  more  seconds 
before  the  first  packet  is  dequeued,  6  seconds  before  the 
second  packet  is  dequeued,  and  6  seconds  before  the  last 
packet  is  dequeued.  As  described  above,  the  packets  in 
Node  2’s  queue  will  have  the  same  destination  as  the  last 

'We  use  transmission  rate  of  1  msg/sec  for  simplicity  of  illustrating 
our  example.  The  link  capacity  can  be  higher,  but  the  computation 
of  cost  will  still  be  the  same. 
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Q 

L[l],  Predecessor(l) 

L[2],  Predecessor(2) 

L[3],  Predecessor(3) 

L[4],  Predecessor(4) 

IL  2,  3,  4} 

0,1 

(X3 

(X3 

oo 

{2,  3,  4} 

0,1 

1,1 

5,1 

1,1 

(3,  4} 

0,1 

1,1 

5  <  1  -f  «;(e(2,3),l)  = 
1-P(4-P1)  =6 

1  <  1  -f  m(e(2,4),l)  = 
l  +  {2  +  l)=4 

{3} 

0,1 

1,1 

5,1 

1,1 

0 

0,1 

1,1 

5,1 

1,1 

TABLE  I 

Routing  table  of  Node  1  to  the  other  nodes  in  one  cycle. 


Q 

L[l],  Predecessor(l) 

L[2],  Predecessor(2) 

L[3],  Predecessor(3) 

L[4],  Predecessor(4) 

{1,  2,  3,  4} 

0,1 

OO 

OO 

OO 

|2,  3,  4} 

0,1 

1,1 

5,1 

1,1 

(3,  4} 

0,1 

1,1 

5  <  1  -f  u;(e(2,3),l)  = 
l-f5-f6-f6  =  18 

1  <  1  -f  m(e(2,4),l)  = 
l-b3-f6-f6  =  16 

13} 

0,1 

1,1 

5,1 

1,1 

0 

0,1 

1,1 

5,1 

1,1 

TABLE  II 

Routing  table  of  Node  1  to  the  other  nodes,  with  queueing  considerations  in  a  single-encountering  scenario. 


Q 

L(l),  Predecessor(l) 

L(2),  Predecessor(2) 

L(3),  Predecessor(3) 

L(4),  Predecessor(4) 

{1,  2,  3,  4} 

0,1 

OO 

OO 

OO 

{2,  3,  4} 

0,1 

1,1 

5,1 

1,1 

{3,  4} 

0,1 

1,1 

5  <  1  -f  tt;(e(2,3),l)  = 

1  <  1  -f  w(e(2,4),l)  = 
l-b3-f6-f6  =  16 

{3} 

0,1 

1,1 

5,1 

1,1 

0 

0,1 

1,1 

5,1 

1,1 

TABLE  III 

Routing  table  of  Node  1  to  the  other  nodes,  with  queueing  considerations  in  a  multiple-encountering  scenario. 


packet  that  is  just  inserted  into  the  queue.  For  the  packet 
that  is  going  to  Node  3,  the  first  two  packets  queued  are 
going  to  Node  3  as  well.  Similarly,  for  the  packet  that  is 
going  to  Node  4,  the  first  two  packets  queued  are  going 
to  Node  4  as  well. 

Note  that  in  this  time-varying  graph,  Node  2  only 
meets  Node  3  and  Node  4  once,  at  6th  second  and  4th 
second,  respectively.  We  characterize  a  node  as  single 
encountering  if  it  only  meets  another  node  in  the  network 
once  in  the  period  of  the  time-varying  graph.  In  this 
example  of  the  time-varying  graph  with  a  period  of  6 
seconds.  Node  2  is  single  encountering  with  Node  3  and 
Node  4  as  Node  2  only  meets  with  them  once  during 
a  period.  We  will  discuss  the  more  general  case  where 
a  node  meets  another  node  more  than  once  in  the  next 
section. 

Now  the  cost  of  L[v\  should  he  compared  with  L[u]  + 
w{e,  L[u\+T),  where  w{e,  L[u\+T)  =  time{l,e,  L[u]  + 
T)  +  P  X  {Qsize  —  1).  time{x,e,  L[u\  +  T)  is  the 
function  that  outputs  the  time  it  takes  to  deliver  x 
number  of  packets  hy  the  edge  e  formed  hy  Node  u 
and  Node  v  since  time  L[u]  +  T.  P  is  the  period  of  the 
time-varying  graph.  Qsize  is  the  size  of  the  queue  at 


Node  u,  including  the  packet  going  to  Node  v  through 
Node  u.  P  X  {Qsize  —  1)  is  the  time  to  dequeue  the 
rest  of  {Qsize  —  1)  packets.  In  the  example  above, 
w{e,L[u\  +  T)  =  5  +  6  X  2  =  17  is  consistent  with 
what  is  shown  Table  II^. 

B.  Multiple  Encountering 

We  examine  node’s  meeting  another  node  more  than 
once  in  one  period  of  a  time-varying  graph.  In  other 
words,  a  node  meets  another  node  if  and  only  if  there  is 
communication  between  them.  For  example,  in  Figure  2, 
Node  2  meets  Node  3  more  than  once  at  the  4th  second 
and  6th  second.  We  characterize  Node  2  as  one  that  is 
multiple  encountering  with  Node  3  during  one  period 
of  the  time-varying  graph.  If  two  nodes  encounter  each 
other  multiple  times  in  a  period,  certainly  that  the  time 
it  takes  for  all  the  packets  queued  up  in  one  node  to 
be  delivered  to  another  is  less  than  if  the  two  nodes  en¬ 
counter  only  encounter  each  other  once  in  a  period.  Since 

^The  table  for  L[3]  entry  when  Q  =  {3,4}  is  comparing  5  with 

1  -I-  w{e{2,  3),  1)  =  1-1-17  =  18.  The  1  that  is  added  to  17  is  the 
shortest  time  to  reach  Node  2  plus  the  shortest  time  then  from  Node 

2  to  Node  3. 
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Fig.  2.  4-node  time-varying  topologies,  multiple-encountering. 

for  each  encounter  they  have,  packets  can  he  offloaded 
from  one  node  to  the  other.  Note  that  in  a  period,  there 
can  he  both  single  and  multiple  encountering  nodes.  The 
characterization  depends  on  the  node  under  examination 
and  on  the  neighbor  with  which  it  encounters.  The 
example  illustrates  a  general  case  of  computing  the  cost 
of  “L[u]  +  w{e,L[u]  +  T).”  Once  again,  we  assume 
that  there  are  two  messages  in  Node  2’s  queue,  the 
transmission  rate  is  1  msg/sec,  and  propagation  delay 
is  negligible. 

When  Q  contains  {3,4},  L[3]  is  compared  to  1  + 
w{e{2, 3),  1)  =  1  +  3  +  2  +  4.  w{e{2, 3),  1)  =  3  +  2  + 
4  because  it  takes  3  more  seconds  to  dequeue  the  first 
packet,  2  more  seconds  to  dequeue  the  second  packet, 
and  another  4  seconds  to  dequeue  the  last  packet.  Since 
Node  2  encounters  Node  3  more  than  once  in  one  time- 
varying  graph  period,  the  first  two  packets  are  able  to 
be  dequeued  in  one  period.  Thus,  the  number  of  packets 
that  are  dequeued  in  one  period  depends  on  the  number 
of  times  a  node  meets  the  other  node. 

IV.  Evaluation 

We  implemented  and  integrated  delay  tolerant  ca¬ 
pability  to  MARP/MDP  routing  protocol  on  Qualnet 
4.5  [1].  Local  queuing  aware  scheduling  was  also  im¬ 
plemented.  The  following  subsections  provide  the  com¬ 
parison  between  MARP/MDP  and  MARP/MDP-i-DTN 
and  the  comparison  between  MARP/MDP-i-DTN  and 
MARP/MDP-i-DTN-i-QC^,  respectively. 

A.  MARP/MDP  vx.  MARP/MDP+DTN 

In  the  first  experiment,  we  evaluate  the  performance 
of  MARP/MDP  and  MARP/MDP-i-DTN  on  a  realistic 
scenario.  This  scenario  has  a  fairly  complex  topology 
with  nodes  on  varying  paths  and  with  intermittent  con¬ 
nectivity.  A  network  flow  was  configured  to  source  at 
node  GlobalHawk  and  sink  at  node  AWACs  2.  Con¬ 
nectivity  between  these  nodes  is  particularly  sporadic 
and  a  complete  end-to-end  path  is  rarely  available.  This 
configuration  provides  an  excellent  opportunity  to  show 
the  improvement  possible  with  MARP/MDP-i-DTN  over 

^QC  stands  for  Queuing  Compensation. 


Fig.  3.  A  realistic  representative  airborne  network  scenario. 

a  non-delay-tolerant  solution.  Figure  3  shows  the  topol¬ 
ogy  where  the  oval  shapes  are  the  routes  of  the  nodes, 
the  thick  arrow  shows  the  desired  network  flow,  and  the 
dotted  lines  are  current  available  connections,  which  wifi 
change  during  the  simulation  as  the  nodes  move. 

The  simulation  was  run  over  three  experimental  vari¬ 
ables  —  radio  range,  delay  tolerance,  and  flow  volume. 
Varying  the  radio  range  will  change  the  connectivity  of 
the  network  and  the  available  paths.  Varying  the  delay 
tolerance  will  show  the  difference  in  performance  due  to 
protocol  optimization.  Flow  volume  is  used  in  order  to 
illustrate  some  behaviors  of  the  network  which  would  be 
difficult  to  grasp  in  only  one  fixed  trial.  A  low  volume 
flow,  consisting  of  300  byte  packets  sent  one  every  ten 
seconds  for  the  duration  of  the  simulation  wifi  be  used  to 
show  the  overall  route  availability  between  ClobalHawk 
and  AWACs  2,  while  a  higher  volume  flow  of  one 
packet  every  0.75  seconds  is  used  to  show  how  a  more 
realistic  flow  would  behave.  In  simulation  the  volume  of 
the  high-traffic  flow  was  limited  by  available  simulation 
hardware,  since  queues  must  be  maintained  for  each 
of  the  simulated  nodes  and  storing  packets  for  delay- 
tolerant  transmission  has  large  memory  requirements. 

As  shown  in  Table  IV,  delay-tolerance  represents  a 
large  increase  in  delivery  percentage  in  all  scenarios.  The 
improvement  is  as  high  as  52%  comparing  column  1  and 
2.  100%  delivery  cannot  be  achieved  due  to  finite  queue 
space  for  holding  delayed  messages,  signal  attenuation 
leading  to  packet  errors  or  drops,  and  the  network 
topology  not  presenting  any  route  to  the  destination  over 
the  final  few  hundred  seconds  (as  the  destination  is  fully 
disconnected).  The  trade-off  is  that  holding  the  messages 
in  queue  causes  a  large  latency. 

By  fixing  the  range  and  type  (e.g.,  look  at  column  1 
and  3),  one  observes  that  low  flow  has  higher  delivery 
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TABLE  IV 

Packet  delivery  and  latency  as  a  result  of  radio  range,  delay- tolerance,  and  flow. 


Column 

Range  (km) 

Type 

Flow 

Packet  Delivery  (%) 

Latency  (s) 

1 

250 

Delay-tolerant 

Low 

58.33% 

975.56 

2 

250 

Non-delay-tolerant 

Low 

6.67% 

0.23 

3 

250 

Delay-tolerant 

High 

33.31% 

1498.58 

4 

250 

Non-delay-tolerant 

High 

6.61% 

0.23 

5 

300 

Delay-tolerant 

Low 

79.33% 

425.82 

6 

300 

Non-delay-tolerant 

Low 

37.67% 

0.17 

7 

300 

Delay-tolerant 

High 

65.39% 

451.42 

8 

300 

Non-delay-tolerant 

High 

34.93% 

0.14 

Fig.  4.  Network  topology  to  evaluate  queuing  awareness. 


ratio  and  lower  latency  than  high  flow  does.  This  is 
because  the  high  volume  of  traffic  at  each  node  causes 
the  queue  to  he  filled  up  that  further  incoming  packets 
have  to  he  dropped.  Since  the  queue  at  most  of  these 
nodes  is  saturated,  it  takes  longer  average  time  for 
packets  to  arrive  at  the  destination.  Furthermore,  hy 
fixing  the  type  and  flow  (e.g.,  look  at  column  1  and 
5),  one  also  observes  that  high  radio  range  has  higher 
delivery  ratio  and  lower  latency  than  the  low  radio  range 
does.  This  is  because  the  high  radio  range  provides  fewer 
hops  to  the  latency.  This  consequently  improves  delivery 
ratio  and  end-to-end  latency. 

The  improvement  in  delivery  ratio  for 
MARP/MDP-fDTN  is  due  to  the  buffering  of 
temporarily-undeliverable  packets  in  intermediate 
nodes  in  order  to  deliver  them  later.  When  links  come 
up,  a  burst  of  DTN  traffic  occurs  as  queued  packets  are 
advanced  towards  the  source.  This  effect  is  visible  in 
huge  spikes  in  bytes  delivered  in  the  DTN  case  when 
the  data  is  viewed  in  timeslices  based  on  delivery  time. 
The  figure  is  not  shown  for  the  sake  of  space. 

B.  MARP/MDP+DTN  vv  MARP/MDP+DTN+QC 

In  the  second  experiment,  we  evaluate  the  perfor¬ 
mance  gain  from  enabling  MARP/MDP-fDTN  with  lo¬ 
cal  neighbor  queuing  awareness,  queuing  compensation 
(QC).  The  topology  is  set  up  as  shown  in  Figure  4 
where  two  separate  CBR  traffic  flows  originate  from 
Node  1  and  Node  2,  respectively,  to  the  sink  Node  5. 
Furthermore,  the  link  between  Node  4  and  5  goes  down 


for  a  short  duration^  periodically.  MARP/MDP-fDTN’s 
and  MARP/MDP-fDTN-fQC’s  PDRs  and  latencies  are 
compared  with  different  transmission  rates,  ranging  from 
2.048  Mbps  to  1024  kbps.  For  each  transmission  rate, 
20  simulation  runs  were  conducted  to  obtain  the  average 
with  95%  confidence  interval. 

Figure  5(a)  shows  the  aggregate  PDR  from  the 
two  flows  in  decreasing  transmission  rates  for  both 
MARP/MDP-fDTN  and  MARP/MDP-fDTN-fQC.  When 
the  transmission  rates  from  both  Node  1  and  2  are 
high,  the  PDR  for  MARP/MDP-fDTN  is  close  to  0%. 
Only  until  the  transmission  rates  begin  to  drop  to 
204.8  kpbs  that  packets  start  to  be  received  at  Node 
5.  Figure  6,  which  shows  the  difference  of  packets 
forwarded  from  Node  3  and  Node  4,  offers  an  expla¬ 
nation  to  why  MARP/MDP-fDTN  is  unable  to  handle 
high  traffic  volume.  According  to  the  figure,  both  flows 
from  Node  1  and  2  forward  through  Node  3  heavily 
in  MARP/MDP-fDTN.  In  MARP/MDP-fDTN-fQC,  how¬ 
ever,  Node  2  is  able  to  know  the  local  queuing  conditions 
of  Node  3  and  Node  4.  Knowing  that  packets  are  heavily 
queued  up  at  Node  3,  Node  2  then  diverts  its  traffic 
to  Node  4,  thus  increasing  the  overall  packet  delivery 
ratio.  Due  to  space  constraint,  results  for  individual 
PDR  for  both  traffic  flows  are  not  shown.  However, 
MARP/MDP-fDTN-fQC  does  provide  higher  PDR  for 
both  traffic  flows  for  every  transmission  rate. 

Figure  5(b)  shows  the  latency  from  the  two 
flows  in  decreasing  transmission  rates  for  both 
MARP/MDP-fDTN  and  MARP/MDP-fDTN-fQC.  The 
results  clearly  show  the  need  for  local  queuing  aware¬ 
ness.  In  MARP/MDP-fDTN,  since  Node  2  keeps  for¬ 
warding  its  packets  to  Node  3,  at  2.048  mbps  transmis¬ 
sion  rate.  Node  3’s  queue  is  constantly  full.  Incoming 
packets  are  dropped  at  the  fully  saturated  queue.  This  re¬ 
sults  in  the  low  packet  delivery  ratio.  As  the  transmission 

■^The  setup  is  to  demonstrate  that  despite  frequent  disconnectivity 
between  Node  4  and  5,  MARP/MDP-fDTN  will  still  utilize  such  a 
link  to  load  balance  the  network  and  at  the  same  time  minimizes 
latency. 
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Sj  MARP/MDP+DTN 
■  MARP/MDP+DTN+QC 


Transmission  Rate  (kbps) 


Transmission  Rate  (kbps) 


(a)  PDR  CDF. 


(b)  Latency  CDF. 


Fig.  5.  The  cumulative  distribution  functions  of  both  PDR  and  Latency. 


Transmission  Rate  (kbps) 


■  MARP/MDP+DTN 
E  MARP/MDP+DTN+QC 


Fig.  6.  The  difference  in  forwarded  packets  between 

MARP/MDP+DTN  and  MARP/MDP+DTN+QC. 


rate  decreases,  Node  3’s  queue  is  still  constantly  full; 
however,  incoming  packets  are  coming  at  the  rate  helow 
the  packet  servicing  rate  that  they  are  admitted  into  the 
queue  successfully.  This  explains  the  higher  PDR  as  the 
transmission  rate  decreases.  Since  these  received  packets 
experience  long  wait  time,  they  attribute  to  the  higher 
average  latency.  Thus,  there  is  an  increasing  trend  of 
average  latency  as  the  rate  decreases  up  to  341.33  khps. 
From  341.33  kphs  onward,  the  latency  starts  to  drop, 
indicating  that  pumping  packets  out  at  a  rate  of  341.33 
kphs  exactly  saturate  Node  3’s  queue  without  any  packet 
drop. 

Since  MARP/MDP+DTN+QC  is  able  to  divert  traffic 
from  Node  3  to  Node  4  to  achieve  load  balancing.  Node 
3’s  queue  is  never  full.  In  fact,  the  queue  is  always 
kept  at  a  low  level  that  the  average  latency  is  never 
above  1  second.  Figure  6  shows  the  difference  in  packets 
between  packets  forwarded  by  Node  3  and  packets 
forwarded  by  Node  4  for  both  MARP/MDP+DTN  and 
MARP/MDP+DTN+QC.  MARP/MDP+DTN ’s  higher 
difference  than  MARP/MDP+DTN+QC ’s  indicates  the 
wide  disparity  in  forwarded  packets  from  Node  3 
and  Node  4.  While  in  MARP/MDP+DTN,  a  lot  more 
packets  are  forwarded  to  Node  3  than  Node  4,  in 
MARP/MDP+DTN+QC,  packets  are  evenly  distributed 


between  Node  3  and  Node  4,  showing  further  evidence 
of  load  balancing  provided  by  queuing  compensation. 

V.  Conclusion 

The  paper  has  enabled  MARP/MDP  with  delay  tol¬ 
erant  capability,  plus  local  queuing  awareness.  Results 
have  shown  the  benefits  of  delay  tolerance  in  improving 
packet  delivery  ratio.  By  being  aware  of  local  queu¬ 
ing  conditions,  nodes  are  able  to  route  around  traffic 
congestion  and  minimize  their  traffic  delay.  As  future 
work  our  team  will  tune  parameters/routing  metrics 
of  MARP/MDP+DTN+QC  protocol  in  accordance  with 
flight  and  link  data  obtained  from  real  flight  tests  like 
Capstone  II. 
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